Methods for the Classification of Data from Open-Ended Questions in Surveys

Which methods can we use to classify data from open-ended survey questions?

Can we leverage these methods to make empirical contributions to substantial questions?

“[…] introducing various methods of classifying data from open-ended survey questions and empirically illustrating their application. A central research question addressed in this thesis therefore concerns the analysis of (short) text data generated by open-ended survey questions.”

1. introducing readers to the survey methodology of using open-ended questions
   • including historical and modern developments, characteristics and challenges of open-ended questions, types of OEQs (e.g., probing)

2. introducing readers to computational methods available for analysis of open-ended answers
   • manual, semi-automated, fully automated
3. applying various of these available methods in three empirical studies

• fully manual methods require high ressources (time and effort)

• but more importantly, human codings can

  • be biased (Mosca et al., 2022),
  • lack objectivity (Inui et al., 2001),
  • introduce errors when coders misinterpret answers or annotation codes (Giorgetti & Sebastiani, 2003),
  • face transparency issues related to unitization and intercoder reliability (Campbell et al., 2013).

• automated methods offer objectivity and systematicness (Zhang et al., 2022)

• still, issues persist (e.g., transparency) which makes it crucial to test and evaluate methods for the social sciences

• data collection approach: three self-administered web surveys with open-ended questions
• data from three U.S. non-probability samples
• methodology for text classification: supervised ML, unsupervised ML, fine-tuning of pre-trained language model BERT, zero-shot learning

• Background:
  • ongoing debates about which type of trust survey researchers are measuring with traditional survey items (i.e., equivalence debate cf. Bauer & Freitag 2018)
• Research Question:
  • How valid are traditional trust survey measures?
• Experimental Design:
  • block randomized question order where closed-ended questions are followed by open-ended follow-up probing questions

• Operationalization via two classifications: share of known vs. unknown others in associations (I), sentiment (pos-neu-neg) of assocations (II)
• Supervised classification approach:
  • 1. manual labeling of randomly sampled documents (n=[1,000,1,500])
  • 2. fine-tuning the weights of two BERT¹ models (base model uncased version), using the manually coded data as training data, to classify the remaining=[6,500/6,000]
  • accuracy²: 87% (I) and 95% (II)

• Background:
  • recent increase of voice-based response options in surveys due to mobile devices equipped with voice input technologies, smartphone surveys and speech-to-text technologies
• Research Question:
  • Are there differences in information content between responses given in voice and text formats?
• Experimental Design:
  • block randomized question order with open-ended and probing questions
  • random assignment into either the text or voice condition

• Operationalization via application of measures from information theory and machine learning to classify open-ended survey answers
  • number of topics, response entropy
  • plus, response length

• Background:
  • conventional notion stating tha trust originates from informed, rational, and consequential judgments is challenged by the idea of an “affective-based” form of (political) trust
• Research Question:
  • Are individual trust judgments in surveys driven by affective rationales?
• Questionnaire Design:
  • closed-ended political trust question followed by open-ended probing question

• Operationalization via sentiment and emotion analysis

• Transcript-based

  • pysentimiento for sentiment recognition (Pérez et al. 2023)
  • zero-shot prompting with GPT-3.5

• Speech-based

  • SpeechBrain for Speech Emotion Recognition (Ravanelli et al. 2021)

• web surveys can be used to collect narrative answers that provide valuable insights into survey responses

• various modern developments (smartphone surveys, speech-to-text algorithms) can be leveraged to collect such data in innovative ways (e.g., spoken answers)

  • always consider challenges and objectives (i.e., in term of sample sizes and sample compositions)

• computational measures can be applied to classify open-ended answers from surveys in order to inform ongoing debates in different fields, e.g.:

  • Study 1: equivalence debate in trust research (Study 1), Study 3: cognitive-versus-affective debate in political trust research (Study 3)
  • survey questionnaire design (Study 2) or item and data quality in general (e.g., associations, sentiment, emotions) (Study 1-3)

Facilitated accessibility and implementation of semi-automated methods.

• supervised models have been a standard in automated methods, but recent developments of large and general-aim pre-trained models (e.g., BERT) allow less resource-intensive fine-tuning

• For example, using only ~13% (1,000 documents from 7,500 in Study 1) documents for fine-tuning resulted in sufficient accuracy (i.e., 87%)

  • increasing the number of manually labeled documents can help in terms of accuracy (i.e., 92% in Study 1)
    • higher number of manual examples also improves the transparency of results: accuracy vs. transparency trade-off
      • start with simple methods and evaluate (e.g., Study 1, first Random Forest, only later BERT)

Increase in possibilities of fully automated methods (e.g., prompt engineering.

• fully automated methods, such as zero-shot prompting can keep up with fine-tuned versions of pre-trained models (e.g., pysentimiento, Study 2)
  • deciding on a suitable number of manual examples and for a method in general (e.g., fully automated (unsupervised) versus semi-automated (supervised, and finetuning) depends on resources such as expected difficulty, desired accuracy, available time and cost resources